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Abstract — For a single transmit and receive antenna system, a 
new constellation design is proposed to combat errors in the phase 
estimate of the channel coefficient. The proposed constellation is 
a combination of PSK and PAM constellations, where PSK is 
used to provide protection against phase errors, while PAM is 
used to increase the transmission rate using the knowledge of 
the magnitude of the channel coefficient. The performance of the 
proposed constellation is shown to be significantly better than the 
widely used QAM in terms of probability of error. The proposed 
strategy can also be extended to systems using multiple transmit 
and receive antennas. 

I. Introduction 

Most practical wireless communication systems e.g. 3GPP- 
LTE [1], WIMAX |2| use quadrature amplitude modulation 
(QAM) assuming that the fading channel coefficient can be 
estimated up to an arbittary level of precision. Acquiring ac- 
curate channel coefficient estimate [3|, however, is challenging 
problem in practice, and requires sufficient resources, e.g. 
large training period and sophisticated signal processing at 
the receiver. The problem is even more complex at reasonable 
values of signal to noise ratios (SNRs), and more often than not 
there is a significant error in the channel coefficient estimate. 

In this paper we consider a scenario where the estimator 
makes an error in estimating the phase of the channel coeffi- 
cient, but estimates the magnitude of the channel coefficient 
correctly. We assume no error in magnitude estimation, since 
the widely used QAM is robust to error in magnitude of the 
channel estimate, and no new constellation design is required 
for small errors in magnitude estimation. On the other hand, 
with error in the phase of the channel estimate, we show 
that the maximum likelihood detector is an angle detector, 
and constellation points should be as far apart as possible to 
minimize the probability or error. Since the angle separation 
between different points of QAM is minimal, using QAM in 
the presence of error in the phase of the channel estimate 
can lead to large probability of error [4|, (also shown using 
simulations in this paper). With error in the phase of the 
channel estimate, a natural strategy is use phase shift keying 
(PSK) that has the maximum angle separation. PSK, however, 
is most suited for the scenario when no information about the 
channel estimate is available (neither phase nor amplitude), 
and cannot take advantage of the knowledge of the magnitude 
of the channel coefficient. 

To exploit the knowledge of the magnitude of the channel 
coefficient, we propose a signal constellation that is a com- 
bination of PSK and pulse amplitude modulation (PAM). For 




Fig. 1. 8 points and 16 points constellation design. 



designing a constellation with M points, the new constellation 
consists of N concentric circles, with K constellation points 
on each circle such that KN — M. In other words, there are 
K lines with N points on each line, where angle between 
each line is For example see Fig. (fib where different 
configurations for M = 8, and 16, are illustrated. The idea 
behind this new constellation is that PSK is robust to error 
in the phase estimate of the channel coefficient, and one of 
the K lines can be detected correctly even in the presence 
of error in the phase estimate of the channel coefficient. 
Assuming that the correct line has been decoded, using the 
exact knowledge of the magnitude of the channel estimate, 
decoding any point on the line is equivalent to PAM decoding. 
Thus, the decoding strategy is a two-step process, where in the 
first step, one of K lines which minimizes the angle between 
the received signal is decoded, and then in the second step 
the point on the decoded line closest to the received signal 
in Euclidean distance is decoded as the ttansmitted signal. 
This combination of PSK and PAM performs better than 
QAM in terms of error probability in the presence of error in 
phase estimation, and has better transmission rate than PSK. 
We note that the proposed concentric circles constellation is 
known in literature (concentric QAM or APSK) where distance 
decoding is used assuming that channel coefficient is known 
exactly 0, 0. The novelty of this paper is in the proposed 
decoding algorithm that makes this constellation robust to 
phase estimation errors. 

The probability of error for the proposed constellation 
design is a function of K and N. Thus, for a given phase 
error, one needs to find the optimal value of K and N to 
minimize the probability of error, given the total number of 
points M = KN. For example it is easy to notice that 
with large phase error one needs to decrease K thereby 
increasing N, while for small phase errors we can have large 
K and consequently small N. We show that minimizing the 
probability of error is a min max problem over K and N, and 
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finding a closed form solution in general is hard. However, we 
show that using numerical integration and using brute force 
search (since the number of feasible points is not very large) 
we can find the optimal values of K and N, We show that 
there is a significant performance improvement by using our 
proposed constellation design over QAM even for small values 
of phase errors, and an arbitrarily large gain for large phase 
errors. 

The proposed strategy to combat phase estimation errors can 
also be extended to the case of multiple transmit and receive 
antennas as follows. With multiple transmit antennas, unitary 
space-time modulation is robust to channel uncertainty 0, 0, 
(9). To exploit the knowledge of the channel magnitude, the 
columns of the unitary matrices are multiplied with constants 
dj, i = 1, 2, . . . , L, thereby increasing the rate of transmission. 
Once again a two-step decoding is employed. First, the unitary 
matrix closest to the received signal in the chordal distance is 
decoded [8], and then m is decoded that lies closest to the 
received signal in the Euclidean distance. 

An important application of the proposed constellation 
design is in the new paradigm of using low precision analog- 
to-digital/digital-to-analog converters (ADC/DAC) for high 
speed/high bandwidth/high sampling rate transmission |10|, 
irTTI . With very high sampling rates only 2 — 3 bits are used 
for quantization, thereby drastically degrading the quality of 
channel estimate. Therefore with low precision ADCs/DACs, 
employing our new constellation design can improve the error 
rate performance significantly, since it is tailor made for 
combating estimation errors. 

II. Notation 

1) CN(fi, a 2 ) —> A complex normal distribution with mean 
/i and variance a 2 . 

2) N(n,a 2 ) — > A normal distribution with mean n and 
variance a 2 . 

3) C — > Complex signal constellation. 

4) Sft — > Represents the real part of a complex number. 

5) < a, b > — > Dot product between two vectors a and b. 

6) f — > Complex conjugate. 

7) \h\ — s- Amplitude of h. 

8) Q(x) —> Complementary error function. 

9) AWGN— s- Additive white Gaussian noise. 

10) Z + — > Positive integers 

11) E{.} -> Expectation. 

III. System Model 

Consider a wireless communication link with a single 
transmit and receive antenna. The received signal y is 



y 



Phx + w, 



(1) 



where x is the transmitted signal with E{x 2 } < 1, h is the 
channel coefficient, w is CiV(0, 1) distributed AWGN, and P 
is the transmit power. To model the richly scattered fading 
channel we assume that h is CN(0, 1) distributed. Let h be 
represented as 

h=\h\e^p j9 , (2) 



where 9 represents the phase of h. An estimate h of h is used 
by the receiver to decode x from y. In this paper we consider 
the case when the amplitude of h is exactly known at the 
receiver, while the estimator makes an error <f> in estimating 
the phase of h. We assume here that the phase estimation error 
4> is uniformly distributed over [—a, a], and h is given by 

h= |/i|exp j(e+ ^ . (3) 
IV. ML Decoding Rule with phase error in the 

CHANNEL ESTIMATION 

From |T|), given the channel estimate h, the probability of 
receiving y given that x was transmitted is 
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P{y\hx) = E^—^exp 



{ — {y—hx exp^)(y—hx exp^)^ } 



)• (4) 



With z — yh'x', 
P(y\hx) = -M^L ex p { - {vy1 - zexp _J> - zt cxp3 * +^ 2 ^ 2 )h 



2tt 



1 

2a 



= exp {_(y! ' t " s,i{zcxp ~ J< * )}+l ' l|2|a:|2)} , 
2tt 



(5) 

Assuming that \x\ 2 = constant, Vie C(signal constellation), 
P(y\hx) is maximized if 



exp 



(6) 



is maximized over the interval [—a, a]. As integral is a limiting 
case of summation, we can write (5) as follows 

-.71-1 „ {m(yh*x* cxp- j( -- a+rk ' > )} 



lim kS^Z" exp^ 



(7) 



where n — !• oo as k — !• and nk is equal to 2a. Therefore, 
to maximize the above summation, we need to maximize 
9ft(y/itaft exp-^- a+rfe }), V(-a+rfc) e [-a, a], over C, which 
is equal to maximizing < y, hx >. Hence the ML decoding 
rule is 



max < y,hx > 

xeC 



(8) 



Therefore to minimize the probability of error, the angu- 
lar separation between the constellation points should be 
maximized. Note that the angular separation between the 
constellation point in QAM is minimal, and PSK cannot 
exploit the knowledge of \h\. To combat phase errors and 
achieve higher transmission rate compared to PSK, in the next 
section we present a constellation design that is a combination 
of PSK and PAM constellations. The basic idea behind the 
constellation design is that PSK is robust to phase errors in 
channel estimation, while PAM constellation can be used to 
increase the rate of transmission by exploiting the knowledge 
of \h\. 

V. CONSTELLATION DESIGN 

Let C be a complex signal constellation with signal set 
{xi, ■ ■ ■ ,xm} with M constellation points. In our proposed 
constellation design we divide the set C into subsets where, 



Lk = {xik ■ angle(x ik ) = a fe }, i = !,-■■ ,N,k = l, 



(9) 
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Fig. 2. A general PSK-PAM constellation design. 



such that M = KN, where \x ik \ = \x a \, Vfc, I G {1, ••• ,K}, 
\%ik\ = R, Vfc, and i? is also the Euclidean distance between 
two nearest elements of each subset, i.e. |^(j_x)fc| = R+\xik\, 
V k. We define a k as the angle of the subset L k made by 
Xik, V i, with the positive X axis. The constellation design is 
illustrated in Fig. [2] If the total number of subsets are K, and 
afe_i < a k < afc+i,V fc, then the angular difference between 
the subset L k and Lfc+i is given by 5 = jr. With the power 
constraint of E{x 2 } < 1 and M = KN, R can be written as, 



R 



6 



(10) 



We represent our signal constellation as (K, N) pair, where 
K is the total number of subsets, and N represents the 
cardinality of each subset. Fig. [T] illustrates 8 points and 16 
points constellation designs for various values of K and N 
e.g. (4,2), (8,1), (2,4), (4,4), (8,2), etc. 

VI. Decoding strategy 

The ML rule derived in the previous section is under the 
assumption that \x\ 2 = constant, VieC. However, in the 
proposed design this condition is not satisfied. To overcome 
the decoding problem, consider a constellation C* where 



, K. Note that 



and Lf. = {xi k : xi k G L k }, k = 1, • 
the constellation points of C* lie on a circle of radius R. 
The receiver now chooses an element X\d in C* which 
maximizes (JHJ. Because the ML decoding rule is essentially 
an angle detector, if x\d G C* maximizes (|8j, it implies 
that Ld is the most likely transmitted subset. After the above 
intermediate step, the receiver views the decoded subset as 
PAM constellation and decodes the received signal x n d G Ld, 
which is closest to y in Euclidean distance. The decoding 
strategy is therefore a two step process as summarized below. 

Step 1— > Decode the subset using the ML rule derived in 
the previous section. Let Ld be the decoded subset 
where 

d = arg max fc < y, hxi k >, k = 1, • • • , K. 

Step 2— > After decoding the subset Ld (Step 1) decode x n d 
G Ld as the transmitted constellation point if 



n = arg minjy - hx id \,i = !,••• ,N. 



VII. Error Analysis For the Proposed 

CONSTELLATION DESIGN 

To derive the probability of error, let xi k G L k be the 
transmitted signal. The decoding error can occur because of 
the following two events, 

Event 1— >• Subset Ld ^ L k is decoded by Step 1 of the 

decoding strategy. 
Event 2—^ Ld = L k , and point Xj k G L k , j ^ i, is decoded 

by Step 2 of decoding strategy. 
The probability of error can be written as, 



P, 



total 



K 

= E 

N 

i=i,(j'^») 



G L/c 



p[L d = L u \x ik G L k ) + P[Ld = L k \x 



(ID 

In the above equation P(x n d — Xj k \xi k ,Xj k G L k ) is the 
probability of the Event 2 i.e. xj k is decoded given that Xi k 
is transmitted. Thus Event 2 occurs if 



\y - hx ik \ >\y- hxj k \. 
If the maximum phase error a satisfies 

(2iV- 1) 



cos(a) > 



(27V) 



(12) 



(13) 



then the probability of Event 2 is same as the probability of 
PAM decoding error given by the formula 11211 . and 



P 



(x nd =Xjk\xi k ,x jk €L k \ = E h Q(^— =— ), (14) 
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where \i — j\R represents the Euclidean distance between Xi k 
and Xj k - We define p(x nd = Xjk\xik,Xjk G L^j as P^m- 
Remark: If the phase error does not satisfy (13), then re- 
estimate h using the y and the decoded x\d G L d (step 1) i.e. 
h = — , re-estimating h forces the phase error to satisfy (13). 

Using (8 1, PyLd = L u \xi k G L k ^j in (1 1) is the probability 
of the Event 1 i.e. 

P({< y,hx lu > - < y,hx lk >) > 0\x ik G L k ), (15) 

where x\ k G L k and x\ u G L u . Let Xi k = iRexj>> a , x\ k = 
i?exp JQ , i G {1, . . . ,N}, and x lu = i?exp^, then 



< y,hx±k >= < \h\iRvPexp 



,{i(«+«)> 



, |ft.|i?exp 



> 



< w. |/i|i?VPexp 



{j(e+<t>+a)} 



>, 



(16) 



and 



< y,hx lu >= < \h\iRVPexp {ji0+ ^ } 
+ < w, \ h\Rexp 



, \h\Rexp 



{j(0+<t>+m 



> 



> 



(17) 



Subtracting ( 17 1 from ( fl"6| ) we get 

(< y, hxi u > - < y, hxik >) =\h\ 2 iR 2 vP(a -b) + w R \h\(c - d) 

+ Wl \h\(e-f), 

(18) 
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where wr and wi represent the real and imaginary part of w, 
and {a, b, c, d, e, /} are defined as follows 

a = cos(0 + p — a), b = cos((f>), c = cos(0 + (ft + (3), 

d = cos(6 + (f> + a), e = sin(0 + <j> + f3), f = sm(d + <p + a). 

Let A = wn\h\(c — d) + wi\h\(e — /). Note that A is Gaussian 
distributed with zero mean and variance N (( c ~ d )^+( e -f) ^ 
since wr and u>j are independent and N(0, 1/2) distributed. 
Therefore 



P( L d = LJxik G Li 



'-\h\cf, 



A > \h\iRWP(a-b) 



With N(p,,a 2 ) distributed A, 



V ax ' 

Hence after some manipulations 

p{h d = L u \x ik G L^j = E\ h ^QW2\h\ 



iR? VP sin 



We define p(L d = L u \x ik G L fc ) as P|^ et . Then P e 
can be represented as 



p total ^ 



K 

E 



Subset 



+ 



N 

E 



r PAM- 



Note that £f =1 



p/c— Vti 
,(u^fe) r Subset 



and Efci,(i#i) P ^4M in 



(23) are primarily governed by P Su 



fc->-(fc+i) 



bset 



and P 



PAM 



i.e. 



PAM 



E h Q 



\h\R 



and 



VIII. Design Principles 



To minimize the probability of error we need to find N and 



(19) 



K that minimizes (24i for a fixed M. Finding TV and K for 
a general case is hard and requires numerical integration and 
brute force search. Next we discuss some special cases for 
which N and K can be found easily. 

• Very small error in phase estimation, small a: In this 
case PpAM ~ -f Suhset- Hence to minimize p* ota ', the 
constellation design should have iV w K. 

• Large error in phase estimation, large a: In this case, 
to minimize pt° tal we need to maximize the angular 
separation 5 between two consecutive subsets. Therefore 
the constellation design should have N > K. 

• Unknown h: When there is no information about the 
channel coefficient, i.e. neither the phase nor the ampli- 
tude of h is known, then PAM decoding is not possible 
and thus N has to be 1. Therefore we get the special case 
of PSK constellation where K = M. 



2 

(20) 

total 



(21) 



the probability of wrong decoding is governed by the nearest 
neighbor of Xi b both in angular distance and Euclidean dis- 
tance. Using and Ppl^ +1) and PgubJlt^ can be 
written as follows 



(22) 



(23) 



IX. Simulation Results 



We now present simulation results to demonstrate the per- 
formance improvement of the proposed constellation design 
over 8-QAM and 16-QAM with phase estimate error. Figs. 
Q and Q compare the symbol error rate (SER) for different 
values of N and K for NK = 8 and 16 i.e. for rate 
3 bits/sec and 4 bits/sec with 8-QAM and 16-QAM as a 
function of SNR for phase error range of (— f , | ) i.e. a = f . 
Simulation results show that for a — -| (large phase error) 
the best constellation to use is (4, 2) for M — 8, and (8, 2) 
for M = 16, since the angular separation with K = 4, 
and K = 8, is |, and j, respectively, which can easily 
tolerate a phase error of |. Our results show that the new 
constellation design achieves a significant SNR gain over 8- 
QAM and 16-QAM constellation. We also consider relatively 
small phase error range of (~fg, fg) in Figs. (|5jl and (|6jl, 
to show the considerable performance improvement of our 
strategy compared to 8-QAM and 16-QAM constellations. 
We also compare the probability of error generated using 
numerical integration and Monte Carlo simulation in Fig. 
where we see that the difference between the two curves is 
minimal. 



Therefore, to minimize the probability of error we need to 
find 



mm max P c , . . 

M \ Subset 

M 



pi->(i+l)s 
r PAM ) 



(24) 



such that {M, — e Z + } and M = KN. 



K 



Hence the constellation design problem (finding N and K) 



requires solving (24i. The difficulty in solving (24i is that it 



is a non-linear optimization problem, and moreover calculating 

P Subs k e? 1} in closed form is difficult. However, P^set^ can 
be easily computed using numerical integration, and we take 
this approach for plotting the analytical results and use brute 
force search to find the optimal N and K. 
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Fig. 3. Performance results for different 3 bits/sec constellation designs for 
a phase error of (— ?)■ 
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Fig. 4. Performance results for different 4 bits/sec constellation designs for 
a phase error of (— ~, ~), 
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Fig. 5. Performance results for different 3 bits/sec constellation designs for 
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Comparison between different constellation designs of rate 4 bits/? 
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Fig. 7. |(a)| Analytic vs Simulation comparison for (4, 2) constellation designs 
for a phase error of (— j^, j^), (b) Analytic vs Simulation comparison for 
(4,4) constellation designs for a pnase error of (— -jjjj jq)- 
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Fig. 6. Performance results for different 4 bits/sec constellation designs for 
a phase error of (— jg, jg). 



X. Discussion 

Under the assumption that there is error in estimating the 
phase of the channel coefficient, we derived the ML decoding 
rule and showed that the widely used QAM is not robust 
to error in phase estimation of channel coefficient. Then we 
proposed a hybrid PAM-PSK constellation, which is more 
tolerant to errors in phase estimate as compared to QAM, and 
has a transmission rate higher than PSK. We calculated the 
error probability for the proposed hybrid PAM-PSK design 
and provided some design principles. Simulation results show 
that our approach results in a significant SNR gain (for large 
phase error) as compared to the QAM constellations. 
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